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ABSTRACT 

This user-oriented evaluation handbook was prepared 
to assist State Advisory Councils on Vocational Education ' (SACVE) in 
their efforts to reipond to various evaluation responsibilities 
prescribed in the 1976 Vocational Education Amendments. The contents 
of this handbook, which include basic steps in conducting a 
jeifciaance audit, are divided into five chapters. Chapter 1 presents 
an introduction to performance auditing end-JUs cusses reviewitg 
programs as an aid to improving them. A description of the- target 
audience, organization of the handbook and\ork sheet number 1 
(assessing resources) are also included in chapter 1. Chapter 2 
provides background information, including a summary of SACVE 
evaluation responsibilities; and evaluation framework for SACVE and 
staff; how to use information gathered by other staff; how to collect 
Dew information through performance auditing: a description of the . 
AIDE (Auditing to Improve Departments of Education) project; and work 
sheet No. 2 (SACVE Information Needs Assessment). Performance 
auditing; is defined in chapter 3, and work sheet No. 3 (Purpose 
Checklist) is presented. The suggested steps in the performance 
atraiting process are detailed in chapter 4, and work sheet NO. 4 
(Audience Check List) is presented. Finally, chapter, 5 discusses how 
to use the results of the audit* The appendix provides an historical 
perspective on the evolution of performance auditing. (Btt) 
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FOREWORD 

The Evaluation Handbook, Evaluation Guidelines a'nd'Practices for'lStato Advisory Councils, 
was prepared to assist State Advisory Councils on Vocational Education (SACVEs) in their efforts 
to respond to various evaluation responsibi lilies prescribed in the 1976 Vocational Education 
Amendments. Recognizing that resources with which SACVEs must work .wilt vary from state to 
state, and that the composition*^ SACVE membership is rich in talent and diversity, National 
Center staff sought contact with many SACVE staff members in an effort to prepare a handbook 
that Is "user-oriented." 

} For their help in describing activities performed by their respective councils, recognition is 
given to: Don Mitchell, Illinois' Joseph W. Parry-Hill, North Carolina; Wallace Vog, New York; 
C. A. Cromer, Nebraska; and Yolanda Lopez, Puerto Rico. 

The performance auditing process is suggested because of its unique potential to serve as a 
SACVE evaluation tool, as an information-gathering vehicle, and as a'meahl through which the 
SACVE may assist program managers in their efforts to improve the vocational education system in 
their states. The methods described are optional, end not intended to.be the "last word" in evalua- 
tion approaches the SACVE may choose to follow. In developing auditing for SACVE purposes, we 
are indebted to the useful suggestions offered by the following individuals who reviewed draft 
copies of the handbook: Alton D. Ice, Texas; Richard Baker, Alabama; Robert White, South Carolina; 
antt Christine Szigeti- Johnson, Maine. 

m k 

To Ralph Bregmen efthe National Advisory Council on Vocational Education we owe a special 
thanks for his review of the draft and his interest and helpful comments throughout the writing 
process. 

The National Center is particularly indebted to Nancy F. Stephens who prepared the handbook, 
and to Janet Spirtr who directed the project. Recognition is also due to Floyd U. McKinney, 
Program. Director and to N.L. McCaslin, Associate Director for Evaluation and Policy, who provided 
assistance throughout this project. In addition, appreciation is extended to Nancy Powell and Carolyn 
Hamilton who typed and editechhe manuscript, respectively. 

QH behalf of the National Center, I want to express appreciation to the Bureau of, Occupational 
and Adult Education, U.S. Office of Education for sponsoring this evaluation handbook. 



Robert €. Taylor 
Executive Director 
♦ The National Center for Research 

in Vocational Education 
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CHAPTER I 
INTRODUCTION 



Members of state advisory councils on vocational education (SACVEs) and their staffs have 
developed \md are refining creative, workable approaches to meeting their responsibilities to 
evaluate vocational education. Expanded and strengthened through the 1976 Vocational Education 
Amendments, these responsibilities have been defined by Congress in such a way as to allow for \ 
. flexible interpretations within the individual circumstances of each state's delivery system. 1 

In presenting this handbook on performance auditing as an approach to meeting SACVE 
evaluation responsibilities, it is not the intention to suggest a definitive evaluation methodology 
which is superior to all pthers. Probably no single approach will neatly resolve the ambiguities and 
conflicts inherent in first interpreting, and then addressing legislative intentions. 



Because a significant amount of program information is already being generated through the 
evaluation efforts of the state boards of education and local and federal agencies, 2 this handbook 
will explore ah evaluative role for SACVEs which would complement rather than duplicate the 
efforts of others: performance auditing. Auditing, whether it investigates financial, compliance, or 
performance aspects of an organization, always involves an examination of already-existing program 
records. If, for example, the state board sets up an evaluation system which produce s information 
regarding program resultr, tfh audit of that set of data Can VerTfy"lheevaluation results," or possibly 
reveal weaknesses in evaluation procedures which resulted in an incorrect program description. The 
SACVE members can then indicate problem areas to the state board without themselves having 
designed and implemented a comprehensive evaluation system— a monumental task which would 
not bepraotical given their many other duties, and which has already been assumed by the state 
board. w 



4 There Is another way in which the performance audit conducted by SACVEs may complement 
iy-exlstlng evaluation activities. Most educational evaluations and audits carry with them e 
"critical" tone which understandably inclines program managers to be somewhat suspicious of 
what they may perceive as "faultfinding" intentions on the part of examiners. Because of the 
advisory nature of the SACV E, there is excellent potential for the SACVE audit to be performed 
In a spirit 6f cooperation with the audited program, thus enhancing the incorporation of audit 
findings into plans for improvement. ' 



See Chepter II for a fuller ditcuuion of 8ACVE evaluation responsibilities. < 

E For further discussion, m Education CommlMion of the 8ff%tEv*lu9tion of Vte#tVona/ Education: Hot*. RnpontfbMftin, a/trf 
R^onm Of tt*t* **d F*d*r*t Ai*Kfm. (Columbus, Ohio: National Canter for Research in Vocational Education, The Ohio 8tett 
University, 10W) • 



Performance Auditing: 
A Complementary Approach 



Reviewing Programs as an Aid to Improving Them 




Target Audience and Organization of this Handbook 

The contents of ^his handbook, which include basic steps in the conduct of a performance 
audit, may be incorporated by SACVEs in several different ways, depending largely on how the j 
auditor or audit team \i chosen. 1 

• If a professionally trained and experlfnced auditor is hired as part of the SACVE^tfrff, 
he/she may benefit from a review of these materials, particularly as they apply to auditing 
within an educational context, and as they stress the assisting and cooperative tone which 
hopefully will differentiate the SACVE audit from other types/ 

• If the SACV E auditor is.already a staff member with little or no experience in auditing, 
this handbook may provide a sound basis for Understanding and practicing the art, as 
well as a springboard for more comprehensive study of performance auditing. 

• In those SACVE organizations in which limited resources are available to hire professional 
staff, the handbook should be useful to members who may use it as a basis for organizing 
an audit team from within their own membership. 

• Finally, as a staff development tool, the materials here can provide a* perspective on the 
performance auditing process which will enhance staff and council members' ability to 
understand the steps involved ih the systematic collection, organization, and dissemination 
of data. 

However, the greatest use of this handbook will probably be by those SACVE staff members 
who may have little experience or training in auditing, but on whom the basic responsibility-for 
designing and implementing SACVE evaluation activities rests. 

It is therefore to the professional and technical staffs of the'state advisory councils that this 
handbook is addressed, in an effort to provide a workable framework for SACVE evaluations and 
an optional nrbrthodology—performance auditing— for staffs to use in obtaining information required 
for purposesyof evaluation, program improvement* and knowledge development regarding the state's 
vocational education delivery system. 

Subsequent cKapters of the handbook will explore the applicability of performance auditing 
to staffs and* members as they seek to fulfill their evaluation responsibilities. Particular attention 
will be focused on how the audit is conducted, with checklists and work sheets accompanying each 
chapter to guide the reader through the auditing process. A discussion of how performance auditing 
evolved from financial and compliance auditing into the process described here, and as is reflected 
in the U.S. General Accounting Office, "Standards,' 4 is included in the appendix. 

/ ' 

Because circumstances vary from council to council, the following work sheet is provided to 
help SACVE staffs assess resources available to them in adopting the auditing process to their own 
unique needs. 
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"Work Sheet No. 1 
Performance Auditing 



SACVE RESOURCE ASSESSMENT 



Completing the following inventory will provide a quick reference of your council's resources. 

f 

1 . What funds are available for the council's yearly activities? 

» 

r 

2. How are those funds allocated? 
Staff 

Projects * ■ \ 

Other 

•»>i » 

_ ;. - " * 

3. What talents exist within the SACVE TfMtfoership? 

Nam* Profession Expertise 



/ 



/ 



I 



• Work Sheet No. 1, cont. 
Name Profession Expertise 



i 



4. Which staff and council members have experience or training in ( 1) accounting, 
(2) auditing, (3) Educational evaluation? (list) 

. s Educational 

, Accounting Auditing - Evaluation 



CHAPTER II. 
BACKGROUND 

State advisory councils on vocational education (SACVEs) were established in 1968 as part p 
of the Vocational Education Amendments (KL 90-576). The intended purpose of these advisory 
councils was summarized in a letter written Aprfl 30, 1973, from Congressmen Albert Quie, William 
Steiger, and Floyd Meeds to the Secretary of Health, Education and Welfare: 

We strongly <believe that independent, lay, advisory councils, at both the national 
and state levels, infusing the view and perspective of business, labor-, and the 
general public Into the educational process, are especially important in the area 
of vocational education. They provide advice to both the educational administrators 
and to Congress which cannot be furnished by professional educators. 

Over the years, conqern for the independence of state advisory councils from their state boards 
has been expressed in Senate and House Reports. Some individuals have doubted that councils 
could make independent evaluations if they were financially dependent and therefore subject to - 
control b\r the administrators of programs they were charged to evaluate. 3 Therefore, amendments 
were insertedin the Vocational Education Amendments ot\1976 (P.L. 94-482) and regulations were 
changed tt>make state advisory councils independent of on-going program operations and adminip 
tration. ' 

One such change found in P. L. 94-482, Section 105 (e), contains the provision for hiring staff: 

Each State Advisory Council is authorized to obtain the services of such / 
professional, technical, and clerical personnel as may be necessary to enable 
it to carry out its functions under this Act and to contract for such services ' j 
. as may be necessary to carry but devaluation functions, independent of y % r 
programmatic and administrative control by other State boards, agencies, and * 
individuals. ^ ^ * .% . J — /- r 

SACVE autonomy was further enhanced by stipulations safeguarding their control- of operational 
funds [(P.L 90-482, Section 105 (f-2)] : 

The expenditure of (these) funds is to be determined solely by the state 
. advisory council for carrying out its functions under the Act, and may not 
, be diverted or reprogrammed for any other purpose by any state board 

agency or individual .... . . 



3 U.8. Congrtst, H6ui t, Report of the Committee on Education and Labor Together with Additional Viawt, The Vocational 
J Education and National Institute of Education Amendments of 1976, 94th Cong.. 2d tew., 1976, H, Rept. 94-1086 to accompany 
H.R. 12836, p. 32. ~f M 



Summary of SACVE Evaluation Responsibilities 

Although the SACVEs have many duties 4 which would require the assistance of full time 
staff, It is the evaluatfon responsibilities that z*re the focus of this handbook. Following arc three 
basic levels of evaluation responsibilities that have been identified in the 1976 legislation by a study 
conducted for the North Carolina Council on Education. 5 



The Review Level: Councils are required to inspect and comment on ^N^. 

• the program evaluations conducted by the st^te board for vocational education 
(P.L 94-482) 

• the distribution of federal funds for vocational education 

» « 
- • the repoVts of -the State Employment and Training Council (established by the 
Comprehensive Employment and Training Act of 1973 as amended) 

The Assisting and Monitoring Level: State advisory councils are mandated to assist-the state boards 
in developing plans for evaluating local programs at least once every five years, and to monitor these 
evaluati6ns. 

.7 v 

The Action Level requires the SACVE to: 

• conduct its own evaluation of the effectiveness of vocational education programs, 
activities, and services ~ 

• prepare an evaluation report which contains recommendations for change as appropriate 

• assess the extent to whifch vocational education and related programs ^present a 
consistent, integrated and coordinated approa9h to meeting the vocational fedudation, 
training, and employment needs of the state's population 

The vagueness of the words "monitor," /assist," and "advise" which occur throughout tfffc^ . 
first two levels of evaluation can be considered advantageous- it allows enough flexibility for the ^ ^ 
SACVEs and state boards to develop the necessary working relationship to perform their respective 
mandates. 6 It is clear that the legislated evaluation assignments, which are both extensive and 
complex, are intended to assist the SACVE members to becom(ft increasingly informech&bout all 
activities relating to vocational education so they will be able to influence program and policy 
decisiorls. The degree of involvement that SACV Es .will achieve in carrying out their advising, 
monitoring, and assisting roles will depend to a large degree on their knowledge of the goals and 
practices of the vocational, education system of their state, their persuasive usaof information, and S 
the degree of cooperatioVexisting between such agenoies as tfie councils, the state board, and state 
employment services councils. The manner in which advisory council members fulfill their evaluation 
responsibilities on the review, assisting, and monitoring levels will depend on resources and circum- 
stances unique to each state, and may resulj in a variety of satisfactory approaches. 

4 

National Advisory Council on Vocational Education, Resource Book for State Advisory Councils on Vocational Education, p. 63. 

5 MDC, Inc., Final Report: Evaluation Project for the North Carolina Council on Education (Chapel Hill, N.C.: MDC.- 1979), pp. 3,4- 

6 Ibid., p. 3. % < r 
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; The acf/or> level is very pointed ih Its language, and requires written evidence of the councils' 
activity artd conclusions regarding: ' 

* their own evaluation of the effectiveness 'of vocational education programs and 

services , . .•; » 

" * * "/» »^ , ' 

• . their own assessment of the etterd 7 to which vocational education and related programs 

, represent a consistent, integrw^pvand coordinated approach to meeting the vocational 
education, training, and employment needs of the state's population 

• " r • . > 

To meet their extensive evaluation responsibilities on the action level, most advisory^pouncils 
have requested assistance of professional and technical staff in addition to drawing on the diverse 
talent and experience found within the council membership. 

Although the rqview and assisting levels-are not less important in .the scope of assigned SACVE 
evaluation responsibilities, it is the action level which is important ih preparing this handbook. • 
because it requires a.strategy which while it enables the SACVE to effectively perform their duties, 
at the same time has to be realistic iiji light of the limited time and resources most councils can afford. 

An Evaluation Framework for SACVE and Staff? 

Coming from varied backgrounds and cjiverse professions, advisory council members have a 
* basic capacity for good judgment, and an ability to become informed in the vocational education . 
field. It would not be surprising / / > ^^B.yto find only a few members who will have had prior 
experience in either evaluation q^flpf^d^il education. 7 Advisory council members are nevertheless 
expected to make the kind of ju$f§M& |»9de daily as ideas and actions are evaluated against a 
background of knowledge and e^@MB^ Although advisory council members are not themselves 
responsible for making program SnJ^tcy' decisions, they are responsible for providing information 
from their own unique perspectivc^nt will improve the ability of decision makers on local, state, 
and federal levels to make ( pwltiyelHnBw in vocational education. 

Given the broad evaluation tasksj&igned to the SACVE and the limitations they must work 
within as lay advisors, advisory council, rr^pjlfcrs must rely heavily upon their professional and 
technical staffs for information about stattprograms. SACVE staffs must in turn sift through the 
information available, and at times conduct independent investigations of state programs, activities, 
and services in order to provide council rrjembers with data relatipg to such questions as: 

• Is progress being made toward tneeting goals set forth in the state plans for vocational 
education? 

• How s well does available data reflect what is actually happening? 

• * 

• What are the procedures for starting up and eliminating a program in vocational 

education in your state? 

• What are the apparent driving forces behind these procedures? (e.g., are they service- 
oriented or self-serving?) 

; • . How accessible are vocational education programs to those who need*and want them? 



National Adviiory Council, RnourcplBook, p. 31 



• How appropriate are training programs to the wider range of community needs? 



• Are programs coordinated sufficiently to be effective for society but not so tied together 
as to hamper the individual? 

# Do evaluation designs go beyond compliance to assess the quality of programs? 

Using Information fathered by Other Agencies 1 

SACVE staffs are hot the onfy groups seeking information about the effectiveness of 
vocational education programs. Because federal funds to local governments usually pass through 
state administrates, it is not^mcommon for an organization to be reviewed by several auditors or 
evaluators representing the federal government, such as the General Accounting 6ffice (GAO), the 
Department of Education and, on the state level, the state department of education. The focus of 
most of these program-reviews or audits has been to ascertain the audited agency's compliance with 
regulations and their effective use of public funds. The data thus gathered by other agencies and 
obtained by SACVE staffs can be an important resource in deciding what focus their own inquiries 
will have.* 

Collecting New Information through Performance Auditing 

* , 

SACVEs cannot, however, rely just on information gathered by other agencies. Their responsi- 
bilities, particularly those evaluation tasks outlined on the action level, require investigations of 
their own. The selection of performance auditing as a strategy for 1) evaluation, 2) information 1 
gathering and 3) improving programs has been used by the U.S. General Accounting Office, and 
described within the educational context for the Office of Education in a project submitted by the 
State of Alabama in 1974, which culminated in the book Auditing Public Education* 

The AIDE Project % 

The "AIDE Project" (Auditing to Improve Departments of Education), concluded in 1974, ? 
was based on consultations with leading authorities in the fields of auditing and education. It 
consists of state-of-the-art descriptions of auditing, the evolution of auditing concepts, future trends 
in auditing, and their implications for educational programs. Finally, it offers some specific 
recommendations for expansion of educational auditing procedures to go beyond financial reviews 
^to aidjln the improvement of programs. 

Following are several selected recommendations from the AIDE Project that are of interest 
in delineating a potential auditing role for SACVE staffs. 

We endorse broad-scope audits of SE As (State Education Agencies) and LE As 
(Ljgpal Education Agencies) if the objectives of such audits are oriented toward 
'helping' rather than 'policing'. " • \ 



o 

Ptttr L. McMicklt and Qtht Elrod, Auditing Public Education. (The AIDE Project), Montgomery: Alabama Department of 
Education, 1974. 



We recommend that Federal Auditors conduct, at least on an experimental 
basis, educational audits that are of a 'general performance' nature rather than 
a 'compliance/performance' nature. 

We suggest that experimental programs be implemented to test the feasibility 
and potential of (1) auditor specialization in educational audits and (2) audit 
teams which include professional educators. 9 

The research and recommendations of the AIDE Project appear to have broad implications for the 
delineation of an audit role for &ACVE staffs- particularly as performance auditors. 



Having completed the resource inventory (Worksheet 1) f SACVE staff should have a clear idea 
of what they have to work'toith. Worksheet which follows, may be used as a guide to planning 
which evaluation activities rhust be undertaken, and how artd when they will be accomplished. Which 
information nfcwjs might be satisfied through a review of existing records and reports (auditing)? ( 



9 Ibid., pp. 207-209. 
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Work Sheet No. 2 
Performance Auditing 



SACVE INFORMATION NEEDS ASSESSMENT 



Information needs may be established by working through the following list of SACVE activities mandated Congress. 
Which information needs might be satisfied by a SACVE performance audit? 

INFORMATION COMPLETE 



TARGET DATES 



1; Advise the State Board (1) on the development of the five-year 
state plan; (2) annual plan; (3) Accountability report; and (4) on 
. policy matters arising from administration of programs established 
under these documents (Sec. 106(e)(1)] 

PLANNING NOTES 

1 ' 
2. 

" . 3. • ' . - 



Yes 



No 



TARGET DATES 



2. Evaluate vocational programs, services, and activities assisted 
under this Act and publish and distribute the results thereof 
[Sec. 105(d)(2)] . \ " . 



INFORMATION COMPLETE 



Yes 



' No- 1 



PLANNING NOTES 



SACVE INFORMATION NEEDS ASSESSMENT 



Work Sheet No. 2 wnt'd. 



3. Prepare and submit to the Commissioner and the National 
Advisory Council oh Vocational Education an annual 
evaluation report * * 1 

PLANNING NOTES ' 

4 

1. ' 
2. 

. 3. ' ' 

, . 4 . . ; - 



TARGET DATES 



INFORMATION COMPLETE 



Yes 



No 

V 



4. Identify, after consultation with the State Manpower-Service 
Council, the vocational education and employment and 
training needs of the State, and assess the extent to which - 
vocational education, employment training, vocatidnal 
rehabilitation, and other programs assisted under this and 
related acts represent a consistent; integrated, and coordinated 
approach to meeting such needs [Sec. 105(d)(4)(a)! 

PLANNING NOTES ' 



1. 
2. 
3. 



\ 



TARGET DATES 



INFORMATION COMPLETE 



Yes 



No 



-2*4- 



.Hold at (east one public meeting each year during which 
the public it given an opportunity to express views 
concerning the vocational education program of the 
•State [Sec. 1 05(d)] 

• PLANNING NOTES 



;1. 
2 



3. 
'4. 



TARGET DATES 



INFORMATION COMPLETE 



Yes 



No 




Provide technical assistance as requested to local 
advispry councils and recipients of P.L 94-482 fuods 
in the establishment and operation of local advisory 
councils [Sec. 105(q)(t)&(2)] 



PLANNING NOTES 

1. 



J 



TARGET DATES 



INFORMATION COMPLETE 



Yes 



No 



r CHAPTER III. 

PERFORMANCE AUDITING: DEFINED 

f 

Performance auditing as presented m this handbook Is not a "made-up" activity especially 
created for the SACVE, but a set of procedures which has evolved from financial auditing to 
embrace administrative responsibilities for program performance as well. The following definition 
is sufficiently comprehensive and provides a structure within which the major components of 
performance auditing may be observed and discussed: 

Performance auditing is an analytical process consisting of preparation, 
conduct, reporting, and reconciliation. The basic elements of this process 
are: anjjbtiepentient auditor, or team of auditors, who execute the 
process upon an auditee for an audit recipient. The scope, or areas of concern, 
involves program performance, factors such as economy, eff iqjency, and/or 
effectiveness of operational-controls, management information systems, 
prd|bny, etc. The objective or purpose of performance auditing can be some 
corolhation of accountability and management assistance in improving and/or 
communicating program results. 10 ' -. ' 

The General Accounting Office has been a leader in providing audit standards which emphasize 
the examination of the effectiveness of program performance through a review of .results. It is those 
standards and procedures which are of particular interest to us in defining an audit role for the 
SACVE and their staffs, (see Appendix). 

The SACVE, may choose to audit program performance through a review of evaluation data 
produced through the state board evaluation system. This procedure will be referred to as an 
Evaluation Audit. On the other hand, the SACVE Performance Audit may involve direct contact 
with the audited programthrough interviewing, observation, and review of pertinent materials. 

This procedure will be designated a Ganaral Program Audit. In summary: 

; ' ,. ■ . ' : ' ' •.. 

i ; 11 - - " \ ' ■ ; . 

The evaluation audit - a review which consists only of -the evaluation of previously prepared 

evaluation plans, procedures, and reports of the audited agency. The evaluation audit' may be 

conducted without direct visits, to the audited agency, or it may be incorporated into the 

r general program audit process to provide additional depth to the overview. 



The ganaral program audit - an overview examination of the major control points; in the 
' organization, or an in-depth review of a few areas of the organization. The general program 
audit typically gathers in/ormation through: 

~~~7~"«~ 0rir8ltr>n f sl5^--eJnieJ7i • observation • review of materials 
L— . } i : — >— : ' 

' r . . ■ ' • : . • 

■ . v i r. » 

TO IWd.,pp.34fnd36. 



What is the Purpose of SACVE Performance Audits? 



Although by its very definition a performance? audit implies some degree of accountability of 
the audi tee to the audit recipient, the suggested role for SACVE staffs will de emphasize that aspect 
in favor of aiding "management and improving future operations. Vocational education programs are 
already undergoing numerous evaluations anql audits for accountability /c^i^liance purposes. ~, 

While other agencies are performing audits whose purpose is to ascertain accountability and 
compliance with program standards, such audits are often perceived as threatening by the audited 
agency who may suspect that auditors are looking only for defects rather than accomplishments. 
These circumstartfces tend to discourage an atmosphere of mutual interest in program improvement 
-which should be the final result of all evaluation effort. The SACVEs, however, as independent 
advisors to vocational education, have the unique opportunity to perform audits in a spirit of 
cooperation with program administrators who, in turn, may feel more receptive to a review conducted 
by -a non-funding agency, whose purpose is to help rather than to criticize. 

In addition to (1) providing a vehicle for encouraging program managers to help overcome 
negative associations with "evaluation" (we do not suggest this can be accomplished overnight!), 
the performance audit can \2) yield information important to SACVE members and staffs in 
increasing their general understanding of the vocational education process, and (3) provide a method 
through .which the SACVEs can discharge their evaluation responsibilities as prescribed in the 1976 
Vocational Education Amendments. 

k. 

Who is Involved in the SACVE Pet^prmance Audit? 

Any audit— financial, compliance, or performance— will involve an auditor, an auditee, and an 
audit recipient. Audits are usually performed upon an audited on behalf of an audit recipient to 
whom the auditee is financially or legally accountable. In the case of SACVE audits, this relationship 
will be less formal in/that no vocational education program or service is strictly accountable to the? 
SACVE. However, an audit relationship can be established, with the Advisory Council itself acting • 
as audit recipient of the audit performed by its own staff on any state program, activity, or service 
related to vocational education whose available evaluation data would allow that program's results 
to be reviewed for effectiveness by the SACYE auditors. As we have noted earlier, the advisory 
nature of the SACVE may be advantageous in helping to create a positive climate for the performance 
audit. 

The auditor or audit team*must possess the essential characteristics of independence, competency, 
and professionalism. There are several possibilities in choosing a SACVE auditor: 

• Themjditor may be professionally trained and experienced, hired by the SACVE as a 
permanent Or temporary staff member. He/she may conduct performance audits either > 
alone or with a staff, including SACVE members who have knowledge or expertise 
important to the program being audited. 

• A staff member knowledgeable in vocational education who is self-trained in performance 
auditing techniques may serve as the auditor. The SACVE auditor in this option would 
probably function best as an audit team leader who carefully recruits other SACVE 
members and staff to round out the experience and knowledge requirements for. the audit. 
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• A third possibility would occur in the case where the SACVEs lack resources to employ 
professional staff and must therefore recruit an audit team and leader from their 
membership. The possibility exists that a trained auditor.^ also a SACVE member. 
However, where little experience or training exists, the audit team and leader can prepare 
themselves by thoroughly review Ing performancrmrdlttng steps found In this handbook 
in order to maintain their credibility as competent auditors. 

How is the SACVE Audit Conducted? 

Although the four parts to an audit-preparation, conduct, reporting, and settlement-apply 
equally to financial, compliance, and performance auditing, the steps of- the performance Auditing 
process will be highlighted in detail in the following chapter. The two forms of performance auditing 
recommended to SACV E staff, the general program audit and the evaluation audit, are treated 
during a discussion of the conduct stage of the performance audit, including suggestions as to the 
complementary use of the two processes. 

In deciding which form their performance audit should take, SACVE staffs may be guided by 
the following checklist of audit purposes. A decision to confine the audit activity to a review of 
evaluation plahs and reports may be based on a "yes" answer to the first thrfe. questions. The 
remaining questions may serve as examples of objectives whjch might shapa*the preliminary plan 
for conducting a general program audit. 
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Work Sheet No. 3 
Performance Auditing 






- ■ - PURPOSE CHECKLIST 


Evaluation Audit 


7 


YES 


NO 




□ 


p 


Can the proposed program, activity, or service produce evaluation plans, ' 
procedures, ano reports wmcn may oe auoiieur 


□ 


□ 

I* 


• 

Is information adequate to verify the accuracy of previously conducted 
evaluation? 






Is the available evaluation data sufficient for SACVE information needs? 

/ 

* . 


General Program Audit 


In the event that insufficient evaluation data is available for auditing a particular program, 
the following questions may help define information purposes which may be fulfilled by the 
more comprehensive general program audit. v (■ f * 


YES 


NO 




n 


□ 


Is information needed about the needs and problems which should be 
addressed by theprogram? ^ 


□ 


□ . 


Is information needed about how well the program meets standards of 
excellence for vocational education? 


□ • 


□ 


■ * 

(s information needed about the long-term' Impact of the program on 
participants? , * - 

■ V 


□ 

• 


□ 


Is information needed on how well this program's outcomes compare With 
similar programs? 


□ 


□ 


Is information needed about the cost of the program in relationship to the 
outcomes produced? 


□ 


D 


* 

Are program managers interested in using audit results to monitor and/or 
improve their programs? , . 


□ 

* 


D 


Are other audiences, besiderthe SACVE and program managers, interested 
in the results of the dudit? 

• 


t 




» 

< 
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CHAPTER IV. 
PERFORMANCE AUDITING: PROCESS 

I ' I 

The four stages in the audit process presented here are common to/all three /types of audit: 
financial, compliance, or performance. Because. the scope an<f objectives of audits vary, the auditor 
will need to modify these procedures, based on each situation. The steps within; each audit stage 
are similar for all three types of audit; however, the greatest variation/occurs in the conduct sta( 
This handbook emphasizes the performance audit- particularly that part o*f aperf ormance audil 
entailing a review of program effectiveness. Therefore the stepmeceisary to Conduct the performance 
audit are highlighted here, while those for financial and compliance/ audits are not treated* 

Outlined below are the steps in the audit process - preparation, conduct, reporting, reconcili- 
ation — which are elaborated on in. this chapter. Examples*from /vocational education are provjfded 
as well as checklists and work sheets which may be useful in gu/ding SA0VE staff through the 
audit process. - 



Suggested Steps in the Audit Process 



• i ; 'in. 



Preparation , 

1. Decision to Conduct the Audit \ . 

2. Selection of the Audit Team \ 

> * 

3. Pre-engagement Contact j 

4. Auditor Familiarization 

5. First Draft of the Audit PlaiW' 

6. The Audit Entrance Conference 



7. The Walk-Through / 

V". ' 

8. " Revision of fhe Audit Plan 



Conduct/ 

1. The General Program Audit 

Examination of lijput, Process, and Output through: 

a. Interviews 

b. Observation 

c. Review of records and reports 

2s The Evaluation Audit 

a. Chepking and reporting on the strengths and 
weaknesses of the Evaluation Plan 

b. Verifying the accuracy of the evaluation findings 
and report* * 



\ 



Reporting 

t 1 . Continuous Reporting to the Auditee 
2. Flash Reports to the Audit Recipient 
- 3. The Draft Report 

4. The Audit Exit Conference 

5. The Final Audit Report 

* 

6. Audience Checklist 



1 



: ST 

Reconciliation ^ t 

1. Evaluation of Audit Findings and Recommendations 

2. Joint Agreement on- a Plan of Action 

3. Audit Recipient Review of Corrective Action 

4. Audit Agency Follow-up 
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PREPARATION 



1. Decision to Conduct the Audit 

• 

The decision to conduct an audit will generally be made by the audit recipient (in this case the 
state advisory council members). The general purpose, objective, and need for each audit should be 
clearly, determined before steps are taken to initiate it. Generally, the SACVE request for an audit 
will be made in order to gain information for one or both of U>e following purposes: 

a. to gain a clearer understanding of vocatioriaf education in their own state 
or to compare that system with other models (knowledge development) 

b. to execute specif ic evaluation projects. 

To the following examples of objectives far an audit, the SACVE staff may want to adcl their own. 



Evaluation Projects 



1 



• Examination and review of vocational 
education issues relating to: 

Policy development 
Goals and objective setting 
Plan composition 
Resources allocation 

• Review of service delivery through 
examination of: 

Training 
Testing 

Skill assessment 
Placement 
Support .service 
Related services 



/ 



Knowledge Development 

• An examination of the state's vocational 
education delivery system 

Who is involved 
f o What is the degree of involvement 

How does this system compare with other 
vocational education models and systems 

• An examination of the state's evaluation 
efforts 

What are the types and degrees of 

evaluation activity 
How are they related to the larger system 

of education 
What are the'technical problems in 

evaluation 
How are they being resolved 

• Review of evaluation through an examination 
• 4 ' of issues relating to: 

Program evaluations 
System evaluations 

Haying determined the objectives of the audit, a schedule should be prepare^ allowing enough s 
time for tra auditor to prepare for each audit-usually six months or more in advancd* A general 
guide should be prepared and formally documented in the auditor's working papers. 11 It should 
describe specific preparatory aud&steps which clearly reflect the purpose and objectives of the audit. 
Although the guide should be flexible enough for later revision, it should be thorough enough to 
assure the auditee of the auditor's competence. * 

*** <. X 

11 U.8. Gtntrtl Accounting Offlct, $t*nd*rdt for Auditing of Governmwut Org*nf**tlons, Progrwnt, *nd ActMtlks (WwhiDfltOn, 
O.C- Comptroller Gtntrt! of tht U.S., 1972), p. 12. 
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£. Selection of the Audit ^am 

The team approach to auditing lends itself well to reviewing programs for effectiveness, with a view 
to aiding rather than criticizing their management. Whereas no one auditor may^competent ih 
all areas, deficiencies may be balanced by careful .selection of others on the audit Te^.jjfbr exampte, 
inviting one of the program staff members to participate in the audit would not only n^kVtfc**/ 
auditee more receptive to the aucHt but wouJd also increase the competency of the team. Anoflfer 
source of the audit team might be the SACV€>tself, among whose members will be found individuals 
with particular expertise in some aspect of vocational education or perhaps frith previous auditing 
experience. In any case, the audit team should possess the necessary competencies to adequately 
accomplish the objective of the audit. 

- . -. m 

The audit team sfeould have an experienced leader who will act as spokesperson and make job 
assignments and decisions. \ 



The following example of training and experience which characterize a typical government 
education auditor ? 2 is included here for reference, but is not intended to be definitive or all-inclusive. 



analytical%bHity f (usually demonstrated by training in accounting) 
college courses m management theory and controls 



• - 

• on-the-job audit experiences with a variety of otganiiations 

• some background in educational theory, technique, governmental programs, and 

« compliance issues (this qualification may be compensated for by employing the team 
approach to include at least one member experienced in education) 

3.. Pre- Engagement Contact 

in most cases the audit team leader will contact the highest authority ie agency to be audited 
through a phone call and follow-up letter. The following information should be conveyed in the 
initial contact: * 

• What will be audited . ' " 

• The general purpose of the audit * ~ 

• The names of those assigned to the audit 
When the auditors would like to begin the on-site work 
When the audit should be completed 1 

• What preliminary information the auditors would like to have upon thejr arrival 
(organization charts, job descriptions, evaluation^tjrfffid reports, brochures) 

• A request for working space and equipment - • 

• A request that a liaison person be assigned who can represent the auditee organization 
and with whom the auditor* can discuss audit progress and findings 

• A request that a formal entrance conference be called as earlV as possible to discuss the - 
audit with all involved and interested parties 13 

McMlfl|rt # Auditing Public Educations 167. ' « 

13 IWd.,M4, 
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\ 

4. Auditor Familiarization 

Team members will want to become familiar with the organization to be audited. In addition to 
materials provided by the auditee, it may be desirable to consult the library on such topics as* 
proper organization functions and procedures. Pertinent governmental rules, regulations, and 
procedures also should be reviewed. 

In reviewing programs for effectiveness, the audit team will need a working knowledge of 
good program evaluation practices so it can effectively review the auditee organization's data „ 
collection, analysis, and reporting systems. Old files may be examined for information on previous 
audits of the same agency, and discussions of the prospective audit held with other auditors who 
may contribute valuable information. 



5. First Draft of the Audit Plan 

The audit team leader is ready now to prepare a preliminary audit plan, which should consist of 
audit objectives, possible areas of investigation and audit steps. Preliminary job assignments as well 
as time estimates for each of the audit steps should be included. - 




6. Audit Entrance Conference 

*> . » ■ 

On the first day pf the audit, the SACVE audit team leader should call a conference with the head 
of thejjrganization and all interested and involved parties to the audit. The team leader should lead 
the discussion using the following as a guide: 

• Introduce the members of the audit team 

• Explajn the nature and purpose of the SACVE, and its interest in evaluating vocational 
educational programs 

• Explain the purpose, objective, and scope of this particular audit 

• Explain how the SACVE audit team intends to approach this audit (stressing that the 
philosophy of the audit team is to aid the agency rather than to find "mistakes") 

• Explain the timetable and reporting procedures 

• Ask if there are any areas that managers would like the audit team to look into or if the 
•auditors can help in any way through their conduct and reporting of the audit 14 

7. The Walk-Through K * 

t 

By arranging for an escorted tour of the organization, SACVE staff auditors can increase their 
working knowledge of the organization itself. They may wish to ask a few general questions but 
should refrain from over-eager questioning that could cause defensiveness in the contact official 
or other staff members. The important thing during the walk-through is to be very observant for . 
clues that may warrant further investigation. After the tour the audit team should make notes on 
what they have seen. 

14 IMd., p. 85. 



8. Rtvltlon of tut Audit Wot 



aujif^ i 



This final step of the preparation stage affords an opportunity for the SACVE aujift team to update 
its preliminary audit plan. The objectiyes and audit steps can now be reassessed in light of new 
information gleaned from an analysis of preliminary materials supplied by the auditee and the team's 
own observations. The team is now ready to begin the4$onduct stage of the audit. 
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CONDUCT 



It is important to remember that it will be impossible to thoroughly examine and evaluate 
every aspect of the organization. The audit team will want to settle either for a general review of 
major controls and activities or an in-depth review of a few functions or sections. Their decision 
will be largely determined by the original intent and objectives of the audit as detailed in the preliminary 
plarls and modified by information gathered through the familiarization, conference, and walk-through 
steps. If; in the preparation stage, some major questions were raised, the audit team should include 
those areas in its examination. 

The following section of the handbook will provide guidelines for conducting: 

A general program audit: A general program examination which reviews inputs, process, 
' and output factors involved in the program. 

An evaluation audit: A closer look & program results or output through a review of 
evaluation plans and reports. 

V 

General Program Audit - Conduct Stage 

In gathering data for its examination, the audit staff should always reflect the "helping" 
rather than "policing" approach in order to elicit maximum cooperation. It may help to remember 
that there will always be a difference between ideal conditions and workable, obtainable conditions 
in any organization. Any suggestions will be received more openly which are sensitive to the realities 
of the actual situation. 

In any case, Information sources should include at least the following:, , 

• Interviews - Operating personnel (teachers, managers) may be interviewed to determine 
their role and perspectives in the organization. Program constituents (students, parents, 
employers) may provide valuable information regarding program processes and results. 

• Observation - Through observation of actual processes, the SACVE audit staff may 
compare planned objectives and program descriptions with what actually appears to be 
happening. 

• Review - Pertinent materials should be reviewed, both those which reflect program' 
management (accounting records, policy and procedure manuals, instructional materials) 
and those that reflect program results, such as evaluation plans, instruments, and reports. 

♦ I 

The following questions may serve to guide the SACVE audit team through the conduct stage 
of a general program audit. The questions, which are organized in an input-output-process framework, 
are supplemented by questions directly related to vocational education. The questions are not 
intended to be exhaustive, and the SACVE audit team may want to review these and add its own 
list of Important questions. 
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Input 



Policy 



1. What policies guide the operation of the vocational program? 

2. How were they jtjeveleped? By whom? 

3. To what extent are the policies of similar programs administered by different agencies 
consistent with each other? 



Goal and Objective Setting 

1. Have plans and objectives been determined and set down in writing? Are plans and 
objectives stated in measurable terms? 

4 1 * 

2. How were they determined? By whom? 

3. Wtiat needs assessment data were used? 

4. How were these data collected? 

5. How adequate are the data? 

6. To what extent do the goals and objectives address the totality of the population who 
may benefit from this program? v . r 

7. To what extent do the goals and objectives address the expected employability of 
participants? 

8. What additional funds, facilities, manpower, and information could be made available? 
Should be made available? 

Planning 

1. How is the need for the occupational program established? Measured? 

2. What demographic, economic, employment, and educational data are used to 
formulate plans? 

3. Who is involved in the planning process? To what extent? 
0. Are the plans evaluated? How? 

5. How frequently are the plans modified? Qased upon what reasoning? Criteria? 

6. To what extent are the plans formulated in consideration of the plans for the agency's 
other programs? 



Output 4 

Product _ v 

& i 

1. Are results and accomplishments evaluated? Are they compared against planned 
objectives? 

2. Doeit management know what they want to do? have tot/o? should do? How well they 
are doing it? 

r 

3.. How is employability ascertained? 

4. What instruments are used? 

5. To what extent have employers participated in the process? 

6. What percentage of graduates are employable? Early leavers? 

7. What percentage was planned? • f 

8. What percentage are prepared for available jobs? 

9. What happens to participants who are not employable? * 



Outcome 

V 

1. Whajt*§re the actual results and accomplishments of the program? Could more be 
(ttnplishecl? 






2. eHtrogram completers get j 

3. Oo noncompleters get jobs? 

4. How long does It take? * " 

5. How satisfied are employers? 

6. Are participants employed in occupations that are training related? 

7. To what extent is there a high employment potential for the skills taught? 
& Are participants satisfied with their training? 

9. To what extent are changes In plans and curriculum related to the employment outcome 
of participants? . 
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Proems 

Resource Allocation 

1* What are the source and amount of funds allocated to the vocational program?' 

2. On what basis are the funds distributed? 

3. How adequate are, the fiscal, physical, and personnel resources to accomplish the goals 
and objectives for the vocational program? 

4. What are the per participant costs of the program? 

B. How do per participant costs for programs administered by different agencies compare? 

* • 

6. How are resources utilized? (e.g.; are talents and work skills put to optimum use? Are 
machines and facilities employed at their maximum and best ute? Are machines and 
facilities adequate? Would new purchases or changes result in significantly better products?) 

>7. Is there adequate protection and tnaintenance of resources (including personnel)? 

8. Are there any major bottlenecks? 

9. ' Arp*there any unnecessary jobs, equipment, or facilities? 



Operations 

1. What ye the characteristics of the vocational program curriculum? 
a. How are curriculum needs determined? 

*b. How is the curricu I urn. evaluated? j 

c. What special provisions are made for persons with special needs? j 

d. How are special needs persons served? ! 

e. What has been the involvement of employers and crafts councils in curriculum 
develbpment? j 

f. Howis competency ascertained? " j 
Z How does work flow through the organization or division? j 

3. Could, teacher/employee morale, attitude, or behavior be improved? 

4. Are management controls and management information systems adequate? Is information 
complete, prompt, factual, and meaningful? Does management know what is going on?> 
When it is going on? > i 

" ■ ■ fMMi^''^- . ■ : ' j 

5. Art jobs clearly defined? Are' theije j6b performance standards? Are these standards 1 
actually In use? ^ , | 



Evaluation Audit Conduct Stag* 



The program audit examination process previously described may be used by SACVEs yvhether 
or not an evaluation system has been installed by the auditee. However, when the audited agency 
can produce information which has been gathered through a systematic evaluation process— as is 
the case of most statd board programs-the SACVE may choosd to ands program results through 
an examination of evaluation plans and reports as well as the procedures for gathering and analyzing 
tho data. The evaluation audit, which is essentially the evaluation of an evaluation) is appropriate tq 
the SACVE function in several wgys t 

• Most vocational education programs receiving federal funds through the state board of 
education will already be part of the state's system for evaluating programs. 16 As state 
advisory councils are required to participate in the monitoring 16 of these evaluations, by 
performing evaluation audits they would be fulfilling one of their legislated functions 
relative to state board evaluations. 

• The evaluation audit helps to verify the accuracy and reliability of data gathered by 
provfding a check on the evaluation system itself. The results may eliminate the nded to 
conducrthe more time-consuming general program audit. 

• Through their activity %t evaluation auditors, SACVE staff obtain information important 
to SACVEs in carrying out their broad responsibility to make informal evaluative' 
judgments on the effectiveness of vocational education programs. 



Requirements of the Evaluation Auditor. As evaluation auditors, the SACVE audit team 
should exhibit the qualities typical of all good auditors: independence, competence, and profession- 
alism. Adjfltiona.lly,'in order to "evaluate evaluations," at least one team member must be familiar 
with. an&preferably trained in good program evaluation practices. 



j ' Scope of the Evaluation Audit. As noted previously, the evaluation audit may be performed 
without ever visiting the program, activity, or service in question or it may.be incorporated into the * 
general program audit process. The basic functions of the evaluation (tuditor are, briefly, as follows: 17 

Check and report on the strengths end weaknesses o f the evaluation plan, including 

• completeness of the plan 

• validity and reliability of the data collection instruments for the vocational program being 
evaluated 

• internal and external validity of the data collection procedures and conditions 

• . technical quality aVid appropriateness of thfc data storage and analysis techniques 

• cost effectiveness of the evaluation plan 

( w 

^ P.L* 94-492, 90 itau 2187, mc. 112, part A. 
* 1e Ibid., part 8. 

1 7 KSV Adami and Jarry P. Walktr, improving th* Accountobllity of Cf—r Education Prognms (Columbui, Ohio: National Cantar 
for Rt»arch In Vocational Education, Tha Ohio State Unlvarilty, 1979), pp, 28-30. 



Verify the accuracy of the evaluation HfSHUfs and report, Including 

• Does the report clearly describes ' 

obiective of the evaluation 
•program content 

purpoieffr^broceduref of the evaluation 
source! ot Information 

• Are Information-gathering instruments and procedures valid and* reliable? 

• Does the report reveal adequate attention to the review, correction, and analysis of data? 
e Are the conclusions justified? , ' y 

• Is the Independence of fhe-evaluators reflected in objective reporting? 

> 

Evaluation Audit Process. 18 The evaluation auditor is typically responsible for: 

• Development of a plan or contract Tor auditing which includes dates of report submission 
■. and what objectives, instruments, and data will-be examined 

e Conduct of the pre-eudit critique-a review of the program evaluation plan to determine 
if it can be audited and to suggest changes in objectives, instruments, data collection 
procedures, and so on / 

• Preparation of a final tfudit report-an analysis of a sample of the program evaluation data 
and an audit Of the final evaluation report of the program 



The following list of criteria 19 can be useful to the evaluation auditor in assessing the 
effectiveness of an evaluation plan. 



18 Kltin dMcribtt tha (valuation audit proem In mora datall through a kit of workihop matarlali. Staphan Klaln at al., educational 
Frograin Auditing Training Hckaga (lot Anflalat: Cantar for tha 8tudy of Evaluation, Unlvartlty of California, 1 97 1 ) . 

l9 Adaptadfrow Jama»R.8aiK»aV»andiii«nH. Nafilaar.A Batli for Datarmlnlng tha Adaquaey of evaluation Datlgnt. <Kafamaroo 
Michigan I Evaluation Cantar, Collaga of Education, Wattarn Michigan Unlvanlty, 1977). 

/ 
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Criteria for Evaluating Plant - 

. • • . >■ 

Scope 

Does the range of information to be provided include ajl significant aspects of the program 
being evaluated? ( *^ 

Relevance > 

Does the information to be provided serve the information needs of the intended audiences? 

<• ' 

Flexibility 

Does the evaluation plan allow for new information needs to be met as they arise? 

Reliability « 

Will the information be collected in such a way that if someone repeated the study they would 
obtain similar bindings? 

Objectivity 

Have provisions been -mede to help eliminate bias in data collection and processing? 

i » ■ f. ■ 

V . ' . ■ . 

Representativeness < 

Will the informetion collected accurately and fairly portray the program? 

< 

Timeliness 

Will the information be provided in time to be useful to the audiences for the evaluation? 

/ 

Pervasiveness / 

Will the information be provided to all who need it? 

Ethical Considerations 

Will the evaluation guarantee confidential fry and protection for those who provide information? 
Protocol * 

Are conventional protocol steps planned for contacting people in the appropriate sequence and 
for following existing policies and procedures? 

• 

Security 

Have provisions been made to maintain the security of the evaluation data? 

Credibility \ 
Dots the design of Jhe evaluation encourage trust in the results by relevant audiences? 




tial '. payoff, will the evaluation be carried out et a reasonable cost? 

., ,-S. J 
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Criteria for Evaluating Report* 



Following is an outNne of topics which should be included in the evaluation report ot a typical 
vocational education program, rfnd may serve as an example of criteria for report evaluations. 



Introduction 

1. Purpose of the evaluation 

2. Questions tb be answered 

3. Audiences to be served 

Description of the Program Being^Evaluated 

4. Documented needs for the program 
6. Subject matter 

6. Objectives 

7. Instructional procedures/program activities 

8. Students (e.g., characteristics, type, and number involved) 

9. Schbdl setting and involvement 

10. Community setting and involvement 

11. Cost (e.g., total and per pupil amount needed for initial start-up and for continuation) 

Evaluation Procedures 

. 12. Overall framework 

13. Information sources (population, sample) 

14. Data cqlfeptlon procedures/research design 

15. Instruments used 

16. Data analysis procedures 

1 7. Limitations, constraints, and possible sources of bias 

* s 

Results v 

18. Program outcomes in relationship to needs of constituency 
* 19. Attainment of objectives 

20. Unintended outcomes and social benefits w 

21. Cost-effectiveness * 

Interpretation 

22. Value of the outcomes 

23. Conclusions * 

. < ■ A 

Summary 

20 AdMM and Wilktr, Improving Accountrilllty, p. 83. 
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Combining the Evaluation Audit and Program Audit. The two forms of performance auditing 
described in this chapter-the general program audit and the evaluation audit -may be used by the 
SACVE staff in complementary ways. 

If; for example, weaknesses in the evaluation plan or reports are discovered through the 
evaluation audit, the SACVE will have the bpportunlty t<5 suggest changes in the evaluation system 
under review. On the other hand, an evaluation plan and report, in all respects well executed, may 
very Hkely suggest problem areas and questions worthy of further investigation by the SACVE staff. 
They may at this point decide to engage in the full four stages-preparation, conduct, reporting, and 
settlemcfnt-of a general program audit. 

A third possibility is that an audit of evaluation plans and reports may reveal nq problem with 
either the evaluation system or the program, which would signify no need for further investigation 
as long as the original objectives of the audit are satisfied. 



r' 
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REPORTING 



Next to providing information to SACVE members, the most important objective of the 
performance audit activity is to aid program managers in their efforts to improve programs. The 
manner in which audit findings are communicated will affect the use of the audit findings. The 
following steps will help to increase the use of the audit findings. 

1. Continuous Reporting to the Auditee 

It is very important that the §ACVE audit team keep the agency contact official informed 
throughout the audit and especially whenever a significant finding or major question arises. To avoid 
secretiveness, it is best to .handle this situation by arranging a formal meeting with the contact official 
whenever, during the course of the audit/it occurs. By avoiding the situation in which the auditee 
is the "last to know," the SACVE audit team will be promoting their role as supporters and playing 
down the policing aspect of the audit function. 

2. "Flash Reports" to the Audit Recipient 

Upon conclusion of the audit and before a draft report is delivered, it may be appropriate in 
some instances to notify the audit recipient (SACVE) of major findings. Thisreport which may be 
oralw written (one page) should briefly explain the auditors' impressions of a problem, its cause, 
and possible solutions. Or, the flash report may not describe a problem at all, but simply be a short 
Nummary of a specific piece of information requested by the SACVE. 

For example, the SACVE might have asked their staff auditors to take inventory of the age, 
condition, and cost of the office machines currently in use in an bffice practice training program. 
This information could be accumulated, and presented in a ''flash report/' with copies transmitted 
directly tfr the SACVE and to the auditee before a complete report of the audit findings is submitted. 

< ■ , . 

3. The Draft Report 

The draft report should be prepared as soon as possible upon conclusion of the examination, 
and promptly delivered to the contact official well in advance of the audit exit conference. The 
auditee agency must be allowed adequate time to prepare a reaction to the findings. In preparing the 
draft report, the SACVE audit team should note: (1 ) strengths and achievements of the program, 
(2) areas which lack information sufficient for auditing, (3) areas of perceived weakness, and (4) 
conclusions and recommendations for improvement. 

Since the SACVEs are not under contract to a funding agency, nor are they directly responsible 
for policy decisions, they are in a good position to offer positive suggestions in a non threatening 
manner. To be most effective and helpful, the draft report should reflect this position. 



4 T|m Audit Exit Conference 

'The purpose of the audit exit conference, the counterpart of the audit entrance conference, 
is to formally explain the audit findings to the auditee. The same individuals should be in attendance, 
with the possible addition of representatives of the SACVE as audit recipients. The exit conference 
should belled by the SACVE audit team leader, who should: 

• Note management achievements as well as management problems. 

• In a positive manner, carefully review each finding - including supporting logic, 
evidence, probable causes, and suggestions for solutions. 

• Solicit auditee concurrence on each finding. If the auditee disagrees with the auditors' 
conclusions, the auditor should ask for a formal position statement from the auditee for 
inclusion in the audit report. 

• Review with the auditee his/her general observations and impressions and any suggestions, 
he/she feels may be helpful to the auditee but which do not warrant inclusion in the 
audit report. 

• Express appreciation for assistance and courtesies extended by the auditee during the 
course of the audit. 

• Maintain a formal record of comments and communications made during the conference. 

In particular, the auditee representatives should be asked whether they feel the report is 
accurate, fair, and fully includes the position of the auditee organization. The audit team leader 
should carefully consider their comments and determine if any changes, deletions, or adjustments 
should be Jmade in the final report. 



5. Final-Report * 

After the audit findings have been discussed in "flash reports," the draft report, and during 
the audit exit conference, any changes, deletions, or adjustments should be incorporated into the 
final report. If the auditee strongly disagrees with the conclusions, a statement should be solicited 
and included with the final audit report. 

* 1 i * ■ 

Overall appearance can have a surprising impact on whether or not the findings are seriously 

reviewed. The final report should have a professional; attractive appearance. Choose type styles, 
paper quality, multiple colors, and cover designs to highlight important points. The use of photo- 
graphs, charts; and diagrams also should be considered to summarize and/or draw attention to 
portions of the report. 

In general, the audit report structure should include the following: 

• x A short summary, one to two pages, fn the front of the report (indexed to the body 
f of the report) : - 

• The scope and objectives of the audit . r .. 

• Achievements noted during^the course of the audit 

• Problems uncovered by the audit 
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• The) significance and magnkude of the problems 

• Tha cauaai of theta problems 

• Suggested solutions or preventive action 

• Audi tee positions regarding these recommendations, without auditor rebuttal; as well as 
aiy steps that the auditee has already taken to correct the problem " 

% Overall conclusions 



When f inalized v the audit report will provide information for (1 ) improvement of the audited 
program, and <2fr the SACVfl as it seeks tq evaluate the effectiveness of the state's program and 
services. However, there may be other audiences with whom the results may be shared. Worksheet 4 21 
may belised to help the SACV£ Identify additional Interested audiences and their specif Ic Information 



needs. 



I 



r 



21 



Ibid., Improving thi Accountability, p. 34 
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Work Sheet No. 4 
Performance Auditing 



AUDIENCE CHECKLIST 



Instructions: In the first and second columnsiheck ( ✓ ) whether the audit should serve each of 
the following audiences. In th/third column rank the relative importance of each 
audience checked "yes." Use/'one" as the highest rank. In the right-hand column 
list the information needs of the audiences checked ''yes." 

Who are the relevant decision makers a^d opinion leaders who should be considered? What are 
their information needs? 

Information Needs 



Yes 


No 


Rank 




□ 


□ 


□ 

> 


Local school boards? 


a 




□ 


Local advisory committees? 


□ 


□ 


□ 


Community leaders? 


□ 


□ 


□ 


State boards of education? 


□ 


□ 


□ 


Federal funding sponsors? 


a 


□ 


□ 


State advisory councils for 
vocational education? 


Q 


□ 


1 □ 


Legislators? 



lERIC^ 



Are program staff and participants involved in planning the audit? Yes? No? How 
Important is their participation? Rank them. What are their information needs? 





□ 


□ 


• Program staff (teachers, coordinators, 
administrator)? 








Student Participants 


□ 


b 


■ a 


Others involved in implementing the program 
(volunteers from business, industry, labor, 
the professions', or government)? 



Will the public be kept Informed of the goals and results of tHudit? Yes? No? How 
important Is it that they be kept informed? Rank them. WhaYare their information needs? 



□ 
□ 



□ 

□ □ ID: 



□ 
□ 



Parents? 

Other concerned citizens in the local 
community? 

Professional colleagues in vocational education 
and evaluation? 
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RECONCILIATION 



I 

Reconciliation of the audit is the last step of any audit process. Since reconciliation takes 
plaoe between trie audit recipient and the auditee and it may or mqy not involve the direct 
participation of the auditor, it is often neglected in the auditing literature. 22 

Assuming that the SACVE will hope to see some action taken on audit recommendations 
produced through their staff audits even though their relationship to the auditee is not one of 
funding source, requiring strict accountability, the steps to the reconciliation stage are listed here, 
with the expectation that the SACVE may modify them substantially to reflect their advisofy role 
to vocational education programs. <. 



1. Evaluation of Audit Findings and Recommendations 

Careful review of tKe' audit findings and recommendations should be undertaken by the audit 
recipient, in this case the SACVE, in consultation with the sfaf f audit team. 

f 



2. Joint Agreement on a Plan of Action 



The audit recipient should attempt to Ak with the auditee to establish a plan for corrective 
n and a timetable with a series of interim deadlines. 



action 



3. Audit Recipient Review of Corrective Action 

The audit recipient should monitor the status of corrective action at each successive deadline. 

i 

J 

4. Follow-up Audit 

Finally, the audit recipient may want to authorize another audit, to include a review of the 
prior findings. / 9 



Regardless of the extent to which the reconciliation stage is developed, the^SACVE audit staff 
should keep a copy of the completed audit on file for possible future reference. 



22 McMicklt, Avrf/fW rt«Wte education, p. 97. 



CHAPTER V. 
USING THE RESULTS OF THE AUDIT 



The following questions may be useful guides to SACVE members and staff in identifying the 
factors which impact on the implementation of evaluation and audit results. 23 

Will audit findings be used as a basis for making decisions about a program, service, or activity? 

Evaluation is not an end in itself, but should be used as part of policy planning and 
management. Problems may result not only when audit findings are disregarded but also when the 
audit results are misused. Audit or evaluation information is prevented from influencing decisions 

by numerous kinds of misuse such as the following three frequently identified types: 

« 

• Shuffling reports and recommendations back and forth by asking for signatures and 
clarifications has been known to preveht implementation of recommendations. 

• Evaluation results may sometimes be used to defend the status quo against innovation. 

• When evaluation results are accepted uncritically, policy decision* may be based on 
evaluations that used unsound methods or faulty conceptualization. 

Can the audit results be used for public relations or lobbying? 
Can the audit results be used: 

• to keep constituents informed of the status and outcomes of vocational education 
locally, statewide, or nationally; ' 

• to gain support for a particular program or for vocational education in general; 

< ■ , ■ , ' 

** to influence future decisions and legislation not directly related to the single program 
being evaluated? 

The results of an audit can be used as a defense of the status quo or as an impetus for 
change. Listed below are some guidelines for communicating evaluation results to constituents such 
as parents, boards of education, advisory councils, and legislators. 

• Illustrate the outcomes of the program with examples of how individual students, 
teachers, and other participants benefited from the program. 

• Compare the successes and weaknesses of the program to those of other programs. 

• ., Stay abreast of the various political developments that influence the program. 

23 M*m, Improving tf># Accountability, pp. 00-95. 



An the audit recommendations written in a manner that wit/ enhance their significance and 
facilitate their use? 

Evaluation recommendations are more likely to be implemented when: 



# 



they are not overly time-consuming to implement 

they are not overly costly to implement 

they are flexible and present alternative recommendations 

they call for incremental change rather than total revamping of the program 

they place ttje program staff in a central role yet do not make unreasonable demands 
oh 



they are short and concise 
they are easily communicated 



How can some people's resistance to acknowledging audit findings be reduced? 



Some people inherently resist using data to make decisions. Thus, it is often difficult to 
change intuitively-oriented decision-makers into more pragmatic ones. If the SAC VE members 
encounter serious resistance to their performance audit efforts, the following suggestions may help 
to resolve the conflict. 

• Appraise the amount and focus of resistance and consider restating the audit findings 
in a more diplomatic manner 

• Use advisory groups who represent opposing factions to give fair representation of all 
facets of the problem issue 

• Compromise and settle on partial change if necessary 

• Use listening as a deliberate approach; allow persons to express their resistance 

• Communicate or share the benefits from implementing the audit results with persons 
concerned 

• Seek and use direct and personal support from the top of the organization 




\ 

The SAC\)e Challenge - Some Concluding Thoughts 

In suggesting alternative solutions to problems of vocational education, SACVE members and 
staffs are challenged by the following tasks: 

• deciding what they believe the needs of vocational education are 

• understanding what services already exist a 
determining how programs could be altered or changed 
articulating and/or helping to formulate standards for program performance 



# 
# 



on state and local levels 



It is important to remember that there*} little agreement on the standards by which vocational 
education should be evaluated. Yet standarKare clearly being used in every program at every level 
of activity. Perhaps the SACVE may act as a focal pdint for a discussion of standards and values 
which would lead to some agreement about measurable goals and objectives within the stbte system. 



SACVEs have been presented with a formidable challenge: to influence present improvements 
and future direction pf their state vocational education delivery system. In helping to define the 
goals of vocational education, reviewing information to ascertain whether objectives are being met 
and presenting their findings to relevant decision-makers, they will have contributed greatly to the 
overall clarity and quality of service d^ivery in the stqte. 

* ' I 

The work performed by the SACVE staff in obtaining information about the state's vocational 
education system is crucial to the successful performance of the council in these three areas. It is 
anticipated that the auditing procedures presented in this handbook will provide a useful vehicle to 
SACVE staffs in meeting the challenge. 



39 



15 



APPENDIX 
Evolution^of Perfomiance Auditing 

Although audits can ba groupad into a numbar of diffarant classes, none of tham it totally 
discrete. A division of audits into the three categories of financial, compliance, and performance 
J^as introduced by the AIDE Project for purposes of their inquiry and will be carried through fn f 
this handbook for purposes of clarity. The three concepts wlii be discussed in their previously stated 
order, which is reflective of the order of their historical evolution. Although the discussion is brief, 
its intent Is to orient SACVE members and staffs to the historical scope of auditing and to provide 
a basis for understanding the nature of performance audits relative to the broader scope of auditing. 



Financial Auditing 

Auditing as an accountability device has been Iq practice since 2000 B.C. when the Egyptians 
developed a system for checking on the contents of the royaj treasury. 1 It was during the Dark Ages 
in England and Europe, however, that the basis of contemporary auditing was founded. During this 
time and up until around 1840, auditors were usually amateurs honored with an appointment which 
would last only through the duration of the audit. Audits were concerned with two objectives: 
detection of fraud and detection of error. Auditors were expected to "be so prudent and so faithful, 
and so knowing in their business, that they have no need of the teaching about things connected with 
the account'* 

Auditing as a profession in the United States was stimulated by the Securities and Exchange 
Acts of 1933 and 1934 which included a provision for auditing all companies registering securities 
for sale. Auditing up to this time was still concerned first with the detection of fraud and secondarily 
with the detection of error. The auditing process primarily consisted of a detailed examination of 
financial records; however, the difficulty and expense of auditing every transaction of large companies 
provided stimulus for the emerging reliance upon internal control and testing (rather than complete 
examinations). »' 

As auditing practices gradually changed to include more sampling techniques^ so too the 
priorities and objectives of auditing changed. The objectives became: first, to ascertain the actual 
financial condition and earnings of an enterprise; and second, to detect errors and fraud. 3 

Audit standards which were approved by the American Institute of Accountants in 1948 (later 
to become the American Institute of Certified Public Accountants-AICPA) are today considered to 
be the foundation of their financial audits. The most frequently referred to statement of standards 
issued by the A I CPA is Number 33, published in 1961, which delineated general, field work, and 1 
reporting standards. 

* Richard Brdwn^ Htitory ofAceounHng*ndAccount*nt9SEdi\r\burth t 8cotlend: T.C. and E.C. Jack, 1006), p» 76? 

8 Peter L McMlckle and Gene Elrod, Auditing Public Education (The AIDE Project), Montgomery, Alebama Department of Education, p. 7. 

3 Robert H. Montgomery, Auditing Thiory *nd Prietlc* <New York : Ronald Prett Co., 1912), pp. 10-11 . 



Latir, tn 1964, a fourth itenderd of reporting was added: 



(4) The report shall either contai.i an expression of opinion regarding the 
financial statements, taken as a whole, or an assertion to the effect that 
an opinion cannot be expressed. When an overall opinion cannot be expressed, 
the^reasons therefor should be stated. In all cases where an auditor's name is 
associated with financial statement!, the report should contain a clearcut 
indication of the character of the auditor's examination, if any, and the degree 
of responsibility he is taking. 



Compliance Auditing 



LA 



Whereas financial auditing occurs mostly in the private domain and is performed for the 
most part by certified and public accountants, compliance auditing is of greater concern m govern- 
ment, and focuses on administrative adherence to legal and administrative requirements. To the 
extent thet financial and performance criteria are spelled out in laws affecting an agency's operations, 
compliance auditing may also embrace these two areas. 

The expansion of auditing to include administrative compliance in addition to fiscal responsi- 
bility may be traced to the 1940s and a general interest in improving management for government. 
When the General Accounting Office (GAO) was created in 1921 a broader scope of auditing was 
implied which was later reinforced in the Legislative Reorganization Act of 1974: 

The Comptroller General is authorized and directed to make an expenditure 
analysis of each agency in the executive branch of the Government (including 
Government corporations) which In the opinion of the Comptroller General, 
will enable Congress to determine whether public funds have been economically 
and efficiently administered and expended. 4 

* Compliance auditing does not depart from the standards which have evolved historically and 
are reflected In the AICPA Standards. The scope is simply expanded to include a review of adminis- 
trative practices. The emphasis on legal and procedural compliance tends, however, to hinder the 
compliance auditor's ability to develop a cooperative spirit with the auditee, whomay perceive the 
auditor solely as a critic of the program. 

Compliance auditors typically approach "tha agency to be audited with a clear list of major 
compliance categories which are to be checked. For example, a sample of local school districts mav 
be examined to determine satisfactory compliance in such areas as: 

• concentration of government funds 

• design and evaluation of project! 

• procumbent of equipment, materials and facilities 

• fiscal controls and reporting 6 



'ibid., p. 89. , 
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Performance Auditing 

Performance auditing is the newest auditing concept on the horizon. It goes beyond fiscal 
and legal accountability to encompass aspects of economy, efficiency, and effectiveness, carrying 
with it moie potential than either financial or compliance auditing for aiding maiuigbis in their 
effort to improve programs. 

The development of performance auditing builds on concepts associated with "management" 
or "operational'' auditing which are audit activities internal to a company and designed to help 
managers improve the economy, efficiency and effectiveness of their own operations. 6 Auditors 
performing these functions for management are in the service of the company rather than in that of 
another exterhal agency. Because of the excellent results reported by companies employing internal 
auditors, and because of the expanding definition of accountability to include program performance, 
the federal government has gradually adopted many of the objectives of operational and managerial 
audits in conducting its own audits. 

< 

In 1972 the General Accounting Office made a significant contribution toward the establishment 
of performance auditing standards by publishing its Standards for Audit of Government Organizations, 
Programs, Activities, and Functions. Building on the audit standards of the AICPA, these new standards 
extended the scope of governmental auditing to potentially include financial, compliance, economy, 
efficiency, and effectiveness, and are applied to both internal and external audits. 

• . , , , is . . 

Because of their potential to upgrade and to stimulate consistency in auditing activity on 
federal, state, and local levels, the GAO standards are reproduced on the following pages for the 
reference of state advisory council staffs. 



8 Ibid., p. 19. 
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GOVERNMENT AUDITING STANDARD^ 
The General Accounting Office 



General Standards 

1. The full tcope of an audit of a governmental program, function„activity, or organization 
should encompass: 

a. An examination of financial transections, accounts, and reports, including an evaluation 
of compliance with applicable laws and regulations. 

b. A review of efficiency and economy in the use of resources. 

c. A review to determine whether desired results are effectively achieved. In determining 
the scope for a particular audit, responsible officials should give consideration to the 
needs of the potential users of the results of that audit. 

* 

2. The auditors assigned to perform the audit must collectively possess adequate professional 
proficiency for the tasks required. 

3. In matters relating to the audit work, the audit organization and the individual auditors shall 
maintain an independent attitude. 

* » 

4. Due professional care is to be used in conducting the audit and in preparing related reports. 



Examination and Evaluation Standards 

1. Work is to be adequately planned. 

2. Assistants are to be properly supervised. 

3. A review is to be made of compliance with legal and regulatory requirements. 

4. An evaluation is to be made of the system of internal control to assess the extent it can be 
relied upon to ensure accurate information, to ensure compliance with laws and regulations, 
and to provide for efficient and effective operations. 

» 

6. Sufficient, competent, and relevant evidence is to be obtained to afford a reasonable basis 
for the auditor's opinions, judgments, conclusions, and recommendations. 

Reporting Standards ~ 

1 Written audit reports are to be submitted to the appropriate officials of the organizations 
requiring or arranging for theeudits. Copies of the reports 4 should be sent to other officials 
who may be responsible for taking action on audit findings and recommendations and to others 
responsible or authorized to receive such reports. Copies should also be made available for 
public Inspection. »■ 

■■ .. ; v " ' • ,." ... • .„*, v .< ; , 

2. Reports are to be issued on or before the dates specified by law, regulation, or other arrange- 
ment and, in any event, as promptly as possible so as to make the information available for 
timely use by management and by legislative officials. 



3. Each report shall: 




a. Be at concise as possible but, at the sanm^ime, clear and complete enough to be 
understood by the users. 

b. Present factual matter accurately, completely, and fairly. 

c. Present findings and conclusions objectively and in language as clear and simple as the 
subject matter permits. 

d. Include only factual information, findings, ancf conclusions that are adequately supported 
by enough evidence in the auditor's working papers to demonstrate or prove, when called 
upon, the basis for the matters reported and their correctness and reasonableness. Detailed 
supporting information should be included in the report to the extent necessary to make 



e. Include, when possible, the auditor's recommendations for actions to effect improvements 
in problem areas noted in his audit and to otherwise make improvements in operations, 
information on underlying causes of problems reported should be included to assist in 
implementing or devising corrective actions. 

f. Place primary emphasis on improvement rather than on criticism of the past; critical 
comments should be presented in balanced perspective, recognizing any unusual difficulties 
or circumstances faced by the operating officials concerned. 

g. Identify and explain issues and questions needing further study and consideration by the 
auditor and others. 

h. Include recognition of noteworthy accomplishments, particularly when management 
improvements in one program or activity may be applicable elsewhere. 

i. Include recognition of the views of responsible officials of the organization, program, 
function, or activity audited on the auditor's findings, conclusions, and recommendations. 
Except where the possibility of fraud or other compelling reason may require different 
treatment, the auditor's tentative findings and conclusions should be reviewed with such 
officials. When possible, without undue delay, their views should be obtained in writing 
and objectively considered and presented in preparing the final report. 

j. Clearly explain the scope and objectives of the audit. 

k. State whether any significant pertinent information has been omitted because it is 
deemed privileged or confidential. The nature of such information should be described, 
and the law or .other basis under which it is withheld should be stated. 

4. Each audit report containing financial reports shall: 

a. Contain an expression of the auditor's opinion as to whether the information in the 



financial reports is presented fairly in conformity with generally accepted accounting 
principles or with other specified accounting principles applicable to the organization, 
program, function, or activity audited. Material changes in accounting policies or 
procedures, their effect on the financial reports, and violations of legal or other regulatory 
requirements, including instances of noncompliance, shall be explained in the audit report. 
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